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2.  Executive  Summary 

This  report  details  the  initial  usability  evaluation  of  the  Post  Mission  Assessment  for 
Tactical  Training  -  Trend  Analysis  (PMATT-TA)  Increment  1  web-based  application  using 
a  novel,  heuristic-based  measure  called  the  Experience-based  Questionnaire  for 
Usability  Assessments  Targeting  Elaborations  (EQUATE). 

PMATT-TA  Increment  1  is  an  online  user  interface  and  database  system  that  primarily 
supports  tracking  data  points  from  live  training  events  for  P-3  and  P-8  anti-submarine 
aircraft.  These  data  points  capture  important  mission  and  contextual  information  (e.g., 
wind  speed,  mission  type,  time,  altitude,  and  communications)  regarding  a  training  event. 
Debriefing  officers  can  review  this  information  along  with  mission  objectives  with 
aircrews  to  identify  performance  strengths  and  areas  for  improvement.  By  centrally 
storing  this  information,  PMATT-TA  also  supports  data  calls  to  understand  fleet 
readiness  and  proficiency.  Additionally,  PMATT-TA  addresses  a  need  for  a  digitally 
based  system  that  streamlines  and  automates  the  process  of  data  collection,  analyses, 
and  feedback  while  also  increasing  accessibility  to  data.  While  the  PMATT-TA  has 
demonstrated  promise  in  supporting  a  reduced  workload  for  debriefing  officers,  it  has  not 
been  subject  to  formal  usability  analysis.  To  be  confident  that  the  system  achieves  its 
aims  and  is  operable  by  end-users  usability  analyses  are  imperative. 

With  usability  goals  for  efficiency,  effectiveness,  and  utility,  the  development  team 
implemented  an  iterative  testing  process  (subsequent  iterations  will  be  reviewed  in 
proceeding  Technical  Reports)  for  the  PMATT-TA  Increment  1 ,  to  evaluate  objectively, 
system  usability.  This  report  documents  the  first  iteration  of  evaluation  using  a  population 
of  human  factors  Subject  Matter  Experts  (SMEs).  It  is  important  to  note  that  these  SMEs 
lacked  domain  specific  knowledge  (e.g.,  knowledge  of  acronyms,  knowledge  of  PMATT- 
TA  system  purpose)  and,  while  six  of  our  nine  SMEs  rated  themselves  as  “Very 
Knowledgeable”  of  usability  evaluation,  only  one  was  distinguished  as  “Expert”.  For  this 
reason,  we  will  refer  to  them  as  “usability  evaluators”  for  the  rest  of  the  review.  The 
current  analysis  is  intended  to  expose  both  general  and  specific  issues  users  encounter 
while  using  the  system  and  to  use  this  information  to  inform  system  developers  regarding 
changes  and  general  system  improvements. 
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Figure  1:  PMATT-TA  Increment  1  Online  Interface 
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3.  Introduction 

While  variation  in  the  literature  exists  when  defining  usability,  one  common  theme  is  the 
extent  to  which  a  human  can  use  a  system  to  achieve  goals  effectively  (Bevan,  1995; 
Law  &  Hvannberg,  2002;  ISO,  1998;  Shackel,  1991).  When  a  system  contains  usability 
problems,  users  will  find  difficulty  operating  the  system  to  complete  tasks  (e.g.,  sign  in  to 
site,  enter  data  into  fields,  post  announcements)  effectively  and  efficiently  (Law  & 
Havannberg,  2002).  The  following  will  outline  three  approaches  to  conducting  these 
assessments:  heuristic  evaluation  (i.e.,  analytical  methods),  usability  survey  approaches 
(i.e.,  empirical  methods)  (Gray  &  Salzman,  1998),  and  hybrid  methods1. 

Heuristic  Approaches 

Nielsen  (1994)  has  referred  to  heuristic  (i.e.,  rules  of  thumb)  approaches  to  usability 
evaluation  as  “discount  usability  engineering  methods”  (p.  152).  Heuristic  evaluations 
require  expert  evaluators  to  provide  their  opinion  of  a  system  according  to  a  set  of 
predefined  rules  (i.e.,  heuristics)  (Nielsen  &  Molich,  1990).  While  initial  heuristic 
approaches  identified  as  many  as  a  thousand  (Mosier  &  Smith,  1986)  rules  to  follow, 


1  These  methods  combine  features  of  traditional  survey  based  usability  methods  with  heuristic-based 
approaches  to  usability  assessment. 
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Molich  and  Nielsen  (1990)  narrowed  the  list  down  to  “nine  basic  usability  principles”  (p. 
249).  The  resulting  approach  to  usability  assessment  addressed  several  issues  afflicting 
traditional  approaches  including  time  to  administer,  cost  of  administration  and 
interpretation,  and  comprehensiveness,  quality,  and  utility  of  information  (Ahmed,  2008). 
In  the  military  environment,  such  efficient  and  economical  methods  are  desirable,  as 
concern  and  debate  over  budgets  continues  to  limit  the  time  and  resources  of  DoD 
personnel. 

Survey  Approaches 

Survey-based  approaches  are  the  most  common  form  of  empirical  usability  assessment 
in  both  industry  and  academia  (Ahmed,  2008).  These  methods  solicit  information  from 
users  that  were  not  involved  in  the  design  of  the  system  (Holleran,  1991).  These  survey- 
based  novice-user  usability  assessments  require  respondents  to  interact  with  a  system, 
and  later  fill  out  a  survey  that  assesses  their  satisfaction  with  the  interaction.  These 
surveys  are  usually  validated  prior  to  implementation,  ensuring  that  they  cover  a  broad 
scope  of  usability  issues.  While  these  methods  are  excellent  for  attaining  the  subjective 
satisfaction  of  users  in  a  relatively  short  time-frame  (Ahmed,  2008),  they  tend  to  fall  far 
short  in  identifying  any  and  all  issues  afflicting  a  system  (Jeffries,  Miller,  Wharton,  & 
Uyeda,  1 991  ).While  no  approach  to  assessing  usability  has  ever  been  completely 
comprehensive,  a  need  for  a  method  that  balances  comprehensiveness,  cost,  time  of 
administration,  and  cross-platform  flexibility  is  certainly  desirable. 

Hybrid  Approaches 

The  EQUATE  is  a  hybrid  approach  that  leverages  a  heuristic-based  usability  assessment 
method  in  a  survey-based  format  that  is  efficient  to  administer  and  analyze,  while 
maintaining  a  necessary  level  of  comprehensiveness  and  flexibility.  The  EQUATE  was 
developed  to  address  gaps  and  issues  with  past  heuristic-based  approaches  while 
integrating  best  practices.  For  example,  while  the  EQUATE  maintains  the  same 
dimensions  and  many  of  the  same  items  that  have  been  validated  in  past  literature 
(Nielsen  &  Molich,  1 990;  Nielsen,  1 994),  it  also  enables  experts  or  evaluators  to  further 
articulate  their  issues  in  a  free-response  format.  Moreover,  the  survey  allows  them  to 
assess  the  severity  of  the  issue  and  provide  suggestions  for  fixing  it.  This  is  important 
because  not  all  features  of  Navy  training  systems  are  imperative  (e.g.,  Graphic  Design  & 
Aesthetics).  In  essence,  the  EQUATE  is  a  single  method  that  takes  advantage  of  both 
empirical  and  analytical  approaches  to  usability  assessment  (Gray  &  Salzman,  1998). 

The  EQUATE  developers  acknowledged  that  both  approaches  have  differing  strengths 
and  weaknesses  and  the  integration  of  the  two  ensures  comprehensiveness  and 
accuracy  of  information.  Additionally,  while  identifying  an  issue  is  important,  without 
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specific  information  Usability  Analysts  must  find  creative  ways  to  communicate  this 
information  to  developers.  Unfortunately,  the  solutions  they  come  to  are  not  always 
sound  in  regards  to  human  factors  and/or  may  still  not  meet  the  needs  of  the  end-user 
population. 

Based  on  the  aforementioned  benefits,  the  EQUATE  was  used  to  provide  both 
quantitative  and  qualitative  feedback  to  system  developers  of  the  PMATT-TA  Increment 
1 .  However,  it  is  important  to  note  that  this  feedback  represents  an  initial  evaluation  of 
the  system  that  was  also  intended  to  provide  the  research  team  with  information  about 
the  EQUATE.  Specifically,  we  are  using  this  analysis  to  better  understand  the 
psychometric  properties  of  the  EQUATE.  Subsequent  usability  and  heuristic  evaluations 
may  be  necessary  in  order  to  achieve  a  desired  level  of  system  optimization  for  end- 
users. 

3.1  Goals 

Ultimately,  the  goal  of  this  effort  is  to  improve  the  PMATT-TA  Increment  1  system  for  the 
end-user  population.  To  accomplish  this  goal  a  novel  approach  to  usability  assessment 
was  conducted.  The  EQUATE  provides  quantitative,  general  feedback  regarding  eight 
dimensions  of  usability  (i.e.,  Error  Handling  &  Feedback,  Graphic  Design  &  Aesthetics, 
User  Interaction  Control,  Memorability  &  Cognitive  Facilitation,  User  Efficiency, 
Learnability,  Consistency,  and  Help)  on  an  easy-to-interpret  1  to  5  scale.  System 
evaluators  using  the  EQUATE  are  also  able  to  provide  free-response  feedback  to  better 
specify  issues,  indicate  severity  and  offer  suggestions  for  fixing  them.  These  free 
responses  were  organized,  reviewed,  and  cataloged  by  the  research  team  in  order  to 
deliver  clear  and  concise  information  to  developers  that  focus  only  on  common  themes 
and  issues  of  high  priority. 

3.2  Objectives 

The  main  objective  of  this  effort  was  to  identify  and  document  issues  afflicting  the 
PMATT-TA  system  and  propose  fixes  to  assist  the  development  team  in  enhancing 
future  versions  of  the  software.  In  addition  to  identifying  specific  fixes  to  components  that 
directly  affect  system  functionality,  the  information  provided  by  the  EQUATE  should  also 
enable  program  developers  to  facilitate  general  improvements. 

An  indirect  objective  of  this  effort  was  to  demonstrate  the  value  of  this  type  of  usability 
method  for  the  consideration  of  use  on  other  DoD  software  systems.  Utilizing  the 
EQUATE  methodology  has  the  potential  to  deliver  huge  cost-savings  by  providing  quality 
feedback  to  system  developers  at  all  stages  of  the  development  cycle.  If  usability 
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problems  arise  after  full  implementation  and  fielding  of  a  system,  fixing  those  issues  can 
be  incredibly  costly  and  practically  difficult2. 


4.  Methods,  Assumptions,  and  Procedures 
4.1  Methods 

Recruitment 

Participants  {N  =  9)  evaluated  the  PMATT-TA  Increment  1,  web-based  application  in  a 
laboratory  setting  with  a  desktop  computer.  Researchers  determined  the  eligibility  of 
participants  and  provided  an  informed  consent  document  to  qualified  participants. 
Participants  then  received  a  brief  explanation  of  tasks  and  measures  as  they  moved 
through  the  protocol. 

Each  participant  filled  out  a  demographic  questionnaire  asking  about  education, 
occupation,  years  working  within  that  occupation,  and  experience  level  within  six  areas. 
The  participants  all  had  at  least  a  Bachelors  degree,  while  the  majority  (n  =  7)  had 
completed  a  Masters.  The  primary  occupation  was  Research  Psychologist  and 
experience  ranged  anywhere  from  1  to  30  years.  Participants  also  reported  their 
exposure  with  Computers,  System  Design,  Software  Development,  Human  Computer 
Interaction  -  Graphic  User  Interface  (HCI-GUI),  Usability  Testing,  and  Heuristic 
Approaches  (see  Table  1 ). 

Table  1.  Participant  Expertise  in  Computers,  System  Design,  Software  Development, 
HCI-GUI,  Usability  Testing,  and  Heuristic  Approaches  {N  =9) 


Area 

None 

Novice 

Average 

Very 

Knowledgeable 

Expert 

Computers 

0 

0 

3 

5 

1 

System  Design 

1 

3 

1 

3 

1 

Software  Development 

3 

3 

3 

0 

0 

HCI-GUI 

2 

1 

1 

4 

1 

Usability  Testing 

1 

1 

2 

4 

1 

Heuristic  Approaches 

2 

2 

1 

4 

0 

2  New  contracts  and/or  Engineering  Change  Requests  are  often  necessary  to  ensure  that  prime  contractors  are  funded  to 
address  all  Change  Requests  and  Deficiency  Reports  when  initial  contractual  obligations  are  completed  prior  to 
identification  of  issues. 
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4.2  Procedures 

Tasks 

Participants  completed  a  series  of  tasks  (see  Appendix  A)  to  replicate  the  collection  of 
post-mission  summary  data  that  will  occur  using  the  web-based  application  PMATT-TA3. 
The  tasks  generated  were  derived  from  an  informal  task  analysis  with  domain  subject 
matter  experts  (SMEs).  The  goal  was  to  generate  a  list  of  tasks  that  are  commonly 
performed  by  subjects.  We  then  selected  tasks  we  felt  could  be  completed  by  usability 
evaluators  with  little  to  no  domain  knowledge.  The  following  is  the  list  of  tasks  we  used 
for  this  study.  Using  hypothetical  information,  participants: 

•  Created  and  edited  a  user  account, 

•  Actively  explored  the  menus, 

•  Created  a  new  event  record  with  provided  information, 

•  Reviewed  a  recent  event  record, 

•  Deleted  an  event, 

•  Explored  options,  and 

•  Created  a  Bravo  Zulu  announcement  recognizing  the  event. 

After  completing  these  tasks,  participants  logged  off  the  system.  While  working  on  these 
tasks,  participants  had  access  to  a  notes  page  or  “heuristic  aid”.  The  aid  listed  each  of 
the  overarching  dimensions  listed  in  Table  2.  The  intention  of  the  aid  was  to  assist  the 
usability  evaluators  while  writing  comments  regarding  the  system. 


Measures 

After  completing  the  PMATT-TA  tasks,  each  participant  was  given  the  EQUATE.  It  was 
developed  based  on  an  extensive  review  of  the  extant  literature  regarding  system  design 
and  heuristic  evaluation.  The  review  identified  a  number  of  items  (i.e.,  200)  that  qualified 
as  design  guidance.  These  200  items  were  then  categorized  and  validated  using  a  card 
sort  with  individuals  containing  extensive  knowledge  or  education  in  human  factors 
psychology.  These  card  sorts  then  informed  the  items  and  dimensions  that  formed  the 
EQUATE.  The  end  product  was  an  expert-led  method  containing  design  guidance  that 
enabled  end-users  to  provide  both  general  and  detailed  usability  feedback.  Likewise,  the 
EQUATE  possesses  the  capacity  to  generalize  to  a  wide  variety  of  systems  (e.g., 
training  and  operational),  potentially  saving  the  DoD  from  further  developmental  costs. 


3  Task  descriptions  included  contextual  information  and  representative  data  for  a  reasonable  amount  of  data  fields  to 
support  completion  of  tasks  without  extensive  domain  knowledge.  This  information  did  not  include  all  required  data  in  order 
to  provide  users  with  the  potential  opportunity  to  observe  error  messages  for  incorrect  data  entry. 
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Each  overarching  dimension  contains  10  to  44  sub-items4  grouped  into  17  sub¬ 
dimensions  (see  Appendix  B). 

Table  2.  EQUATE  Dimensions  and  Descriptions 


Dimension 

Description 

1. 

Graphic  Design  & 
Aesthetics 

Interface  display  elements  (e.g.,  color,  text, 
graphics)  and  layout  support  a  positive  user 
experience. 

2. 

Error  Handling  & 
Feedback 

System  feedback  on  status  and  error  supports 
users’  understanding  of  how  to  interact  with 
the  system. 

3. 

User  Interaction  Control 

Mechanisms  that  allow  the  user  to  feel  in 
control  of  actions  and  system  preferences. 

4. 

Memorability  & 

Cognitive  Facilitation 

System  design  helps  ease  learning  and 
memory  load  (short-term  and  long-term 
memory). 

5. 

User  Efficiency 

System  design  and  functionality  that  supports 
completion  of  tasks  with  minimal  time  and 
effort. 

6. 

Learnability 

System  design  and  aids  support  users  learning 
how  to  use  the  system. 

7. 

Consistency 

System  information  and  actions  are 
consistently  located  and  formatted  throughout 
the  interface. 

8. 

Help 

Readily  accessible  instructions  or  clarifying 
information  that  are  easy  to  use  and  support 
task  completion. 

5.  Results  and  Discussions 

The  quantitative  analysis  is  intended  to  provide  an  overview  of  the  usability  of  the 
PMATT-TA  system  as  a  whole  and  a  localized  evaluation  of  the  usability  of  system  and 
system  components  across  the  heuristic  categories  previously  mentioned  (e.g., 
Learnability,  Help,  and  Consistency). 

The  analysis  revealed  both  positive  and  negative  elements  to  the  PMATT-TA  system. 
Usability  was  assessed  on  a  5-point  scale  (i.e.,  1  =Strongly  disagree,  5  =  Strongly  agree, 
0=  Not  applicable).  Items  on  the  EQUATE  are  framed  in  both  positive  and  negative 
terms  (e.g.,  “The  design  provided  a  pleasant  experience”  and  “There  was  too  much 

4  Sub-items  are  detailed  design  guidance  that  provide  examples  of  positive  or  negative  applications  of  the  overarching 
heuristic. 
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clutter  on  the  display”).  Negatively  framed  items  were  reverse  coded  prior  to  analysis  so 
interpreting  dimension  averages  was  clear.  In  general,  higher  averages  imply  better 
usability  while  lower  averages  imply  potential  usability  issues  exist. 

The  average  across  all  items  of  the  EQUATE  was  acceptable  ( M  =  3.41 ,  SD  =  0.24). 
While  this  average  appears  to  indicate  the  system  maintains  adequate  usability  from  a 
global  perspective,  a  more  detailed  evaluation  of  EQUATE  items,  dimensions,  and  free- 
responses  is  necessary  to  validate  and  elaborate  on  this  assertion.  Only  two  of  the 
heuristic  categories  exhibited  an  average  score  below  3.0,  indicating  the  need  for 
redesign  or  adjustment.  They  were  Learnability  ( M  =  2.89,  SD  =  0.56)  and  Help  ( M  = 

2.81 ,  SD  =  0.93),  demonstrating  participants  (i.e.,  usability  evaluators)  felt  they  needed 
more  help  to  learn  the  system5.  The  remaining  dimensions  all  maintained  adequate  or 
above  adequate  average  scores  (see  Table  3  for  descriptive  statistics).  In  the  next 
section  we  will  review  the  free  responses  for  each  dimension  of  the  EQUATE.  This 
review  will  help  identify  issues  missed  by  the  survey  items,  further  clarify  all  system 
issues,  and  determine  how  one  might  fix  them. 

The  opportunity  for  participants  to  articulate  system  issues  in  a  free-response  format  was 
exercised  extensively.  The  vast  array  of  free  responses  required  the  research  team  to 
qualitatively  and  quantitatively  summarize  feedback  for  the  purposes  of  usability  analysis 
and  reporting.  To  do  this,  researchers  classified  free  responses  based  on  the  heuristic 
dimension  they  most  appropriately  represented;  we  then  poured  through  each  response, 
coding  ‘common  themes’.  Common  themes  were  thought,  by  the  research  team,  to  be 
any  issue  that  at  least  half  of  the  respondents  identified  as  problematic.  In  addition  to 
reporting  common  themes,  we  also  rated  issues  based  on  their  perceived  severity 
concerning  system  functionality.  As  a  research  team  we  acknowledged  that  common 
themes  alone  may  not  be  enough  to  capture  any  and  all  critical  issues  afflicting  the 
system.  The  later  evaluation  was  intended  to  capture  important  issues  not  addressed  in 
the  analysis  of  common  themes.  The  proceeding  overview  organizes  these  issues 
relative  to  the  heuristic  categories  previously  mentioned.  For  further  elaboration  on  the 
free-response  feedback,  see  Appendix  B. 

In  regards  to  Graphic  Design  &  Aesthetics,  more  than  50%  (n  >  5)  of  usability  evaluators 
described  the  following  as  problematic.  First,  latency  between  actions  and  system 
reactions  was  reported  as  troublesome.  Specifically,  one  usability  evaluator  noted  that 
processing  times  made  the  system  unpleasant  to  use6.  Secondly,  usability  evaluators 


5  SMEs  lacked  domain  specific  knowledge  likely  influencing  their  ratings  of  learnability,  suggesting  that  Learnability  be  re¬ 
evaluated  in  future  iterations  of  testing  with  end  users  to  rule  out  this  as  a  factor. 

6  Because  faulty  hardware  or  inadequate  networks  can  cause  system  latency,  additional  latency  testing  should  be 
conducted  by  the  development  team  to  determine  if  specific  hardware  specifications  are  necessary  or  if  network  issues 
affected  the  results  in  this  evaluation. 
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noted  that  grouping,  chunking,  or  aligning  related  fields  would  help  guide  users  and 
organize  related  input.  Lastly,  usability  evaluators  listed  the  lack  of  consistency  in 
entering  time  as  confusing.  As  a  potential  fix,  they  felt  that  “typed”  entry  with  format 
restrictions  (i.e.,  as  a  preferred  method  to  using  slide  bars. 

In  addition  to  these  frequently  identified  issues,  the  research  team  concluded  that  the 
“New  Event”  tab  needed  to  be  more  obvious  to  users.  We  came  to  this  conclusion 
because  it  is  an  issue  that  directly  affected  system  functionality.  Further,  this  issue  was 
identified  by  multiple  usability  evaluators,  and  was  categorized  as  affecting  User 
Efficiency7,  as  a  result,  several  usability  evaluators  needed  outside  instruction  to  start  a 
‘new  event’  dialog.  One  usability  evaluator  suggested  making  the  button  larger  or  using 
color  to  make  it  more  obvious  to  end  users. 


Figure  2:  PMATT-TA  Increment  1  Home  page  “New  Event”  button. 


tseument  for  Tar  tf cal  Training  Trend  Analyslt 


Latest  Events  Latest  Qualifications 


Tele-Off  (Z) 

05/15/2014  04  35 
05/14/2014  20:07 
05/07/2014  09.00 
05/05/2014  0900 
05/05/2014  08  00 
04/30/2014  09  00 
04/282014  09.00 
04/24/7014  AQ  AA 


Event# 

JE 14051  SOI  ASA51S03S 

JC14051401ASAS14I67 

JE14050701T 

JE140S0502T 

JE140S0S01T 

JE14043001T 

/£ 1404 290 IT 

IF14A474A1T 


Draft 

Draft 

Draft 

Draft 

Draft 

Draft 

Draft 


Type 

ASA 

ASA 


Home  Events  Reports  w  Administration  ' 


Name:  Test  Usability 
CAC:  N/A 
Squadron:  N/A 
Wing:  N/A 
Community:  CPRG 


Events  Completed  &  Quals  Passed  by  Squadron  (Last  30  Days) 

Announcements 

it  of  Events  tt  of  Quals 


In  terms  of  Error  Handling  &  Feedback,  more  than  50%  (n  >  5)  of  usability  evaluators 
designated  the  following  as  challenging.  First,  feedback  regarding  which  fields  were 
required  was  only  provided  after  you  hit  save.  This  caused  users  to  take  an  extra  and 
unnecessary  step  in  determining  what  fields  were  mandatory.  One  usability  evaluator 
suggested  that  required  fields  should  be  clearly  labeled  (e.g.,  asterisk)  once  the  fields 
are  opened.  Lastly,  several  usability  evaluators  noted  that  tool  tips  disappeared  from 
view  too  quickly.  Reporting  that  they  were  unable  to  read  the  entire  message  before  it 
disappeared.  The  simple  solution  to  this  issue  is  to  allow  more  time  for  messages  to  be 
available,  especially  when  those  messages  are  lengthy.  Since  estimating  the  amount  of 


7  Researchers  did  not  identify  any  other  common  or  high  priority  issues  when  reviewing  the  User  Efficiency  feedback. 
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time  a  user  needs  to  read  messages  may  be  difficult,  one  option  is  to  leave  the  pop-up 
message  visible  until  the  user  removes  the  cursor  from  the  icon. 

Pertaining  to  the  Learnability  heuristic  from  our  assessment,  there  was  one  common 
theme  discussed.  More  than  50%  (n  >  5)  of  usability  evaluators  reported  that  the  help 
icon  (see  Figure  2)  next  to  the  fields  disappeared  too  quickly  and  were  often  not 
descriptive  enough.  As  an  illustration,  one  usability  evaluator  stated,  “The  “?”  [help] 
entries  were  long  and  needed  to  be  accessed  several  times.  They  weren’t  always 
specific  enough  to  tell  you  how  to  enter  data  either.  For  example,  [they]  didn’t  always 
give  measurement  unit[s]  to  use  (ft,  etc.).”  Evaluators  suggested  that  detailed  help 
messages  should  be  available  to  users  for  a  longer  amount  of  time.  This  is  a  related 
issue  to  that  of  error  messages,  and  therefore,  developers  could  address  both  with  a 
consistent  fix. 

Figure  3:  PMATT-TA  Increment  1  Online  Interface  “Help”  icons. 


While  there  were  no  common  themes  relevant  to  Consistency,  there  were  two  issues 
identified  as  critical  by  the  research  team.  First,  several  usability  evaluators  noted  that 
the  “save/next”  buttons  were  not  always  consistent  from  one  field  to  the  next.  Second, 
one  usability  evaluator  noticed  that,  “when  entering  a  new  announcement,  [the]  drop¬ 
down  list  for  time  entry  is  in  civilian  format  (5:00pm),  but  after  the  selection  [time]  is 
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displayed  in  military  format  (1700).”  The  Evaluator  suggests  that  time  format  remain 
consistent  throughout  the  program. 


Figure  4:  PMATT-TA  Increment  1  Online  Interface  “save/next”  icons 


There  were  no  common  themes  or  critical  issues  identified  for  User  Control,  Help,  and 
Memorability  &  Cognitive  Facilitation.  Furthermore,  and  true  of  the  other  dimensions, 
there  were  many  free  responses  that  noted  many  positive  features  of  the  system  (see 
Appendix  A). 

Discussion 

While  the  quantitative  review  provided  a  broad  glimpse  of  PMATT-TA  system  strengths 
and  weaknesses  concerning  usability,  the  review  of  the  free  responses  allowed  the 
research  team  to  identify  specific  usability  issues  and  propose  possible  solutions  to 
problems.  Overall,  the  system  appears  adequate  in  terms  of  its  usability.  Additionally,  our 
review  demonstrated  that  the  system  has  at  least  as  many  positive  as  negative  features 
as  a  whole,  but  as  with  any  system,  improvements  are  necessary  to  achieve  human- 
computer  optimization. 
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Table  3.  Descriptive  statistics  for  Evaluator  responses  on  the  EQUATE  (A/=  9) 


Dimension 

Range 

M 

SD 

Graphic  Design  &  Aesthetics 

3.34 

3.95 

3.681 

.1968 

Error  Handling  &  Feedback 

3.19 

4.19 

3.506 

.3131 

User  Interaction  Control 

3.20 

4.00 

3.502 

.3161 

Memorability  &  Cognitive  Facilitation 

2.48 

4.15 

3.280 

.5122 

User  Efficiency 

3.16 

4.21 

3.687 

.4324 

Learnability 

2.10 

3.88 

2.897 

.5646 

Consistency 

2.93 

4.50 

3.875 

.5733 

Help 

1.17 

3.91 

2.816 

.9398 

Overall  Usability 

3.13 

3.90 

3.409 

.2486 

6.  Conclusions 

In  sum,  our  quantitative  and  qualitative  analyses  revealed  several  usability  issues 
concerning  PMATT-TA  Increment  1  as  was  consistent  with  the  above  assertion.  In 
particular,  the  quantitative  evaluation  revealed  that  the  dimensions  of  Help  and 
Learnability  might  need  more  attention  from  developers.  However,  some  of  the  lower 
ratings  within  these  two  dimensions  may  be  due  to  the  lack  of  domain-specific 
knowledge  on  the  part  of  the  usability  evaluators.  A  follow-up  study  with  the  end-user 
population  of  PMATT-TA  is  necessary  in  order  to  determine  with  more  certainty  if  these 
dimensions  are  indeed  problematic  and  if  any  additional  issues  might  surface. 

While  the  quantitative  analysis  provided  general  information  regarding  system 
insufficiencies  and  strengths,  the  qualitative  feedback  helped  pinpoint  fixes  and  offered 
suggestions  as  to  how  to  reconcile  them.  In  order  to  consolidate  and  organize  the  free 
responses,  the  research  team  evaluated  each  response  systematically.  The  first  step  in 
this  systematic  evaluation  was  to  identify  common  themes  across  our  nine  usability 
evaluators.  The  second  step  was  to  identify  any  single  free-response  that  had 
implications  for  total  system  functionality  (i.e.,  system  critical  issues). 

The  first  part  of  this  systematic  evaluation  identified  consistent  themes  pertaining  to 
Graphic  Design  &  Aesthetics,  Error  Handling  &  Feedback,  and  Learnability.  In  addition  to 
the  specific  issues  identified,  respondents  also  offered  suggestions  for  fixing  them.  We 
believe  these  suggestions  will  be  especially  informative  to  system  developers.  A  review 
and  specific  examples  of  those  issues  and  solutions  can  be  found  in  the  results  section 
and/or  Appendix  B. 
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Results  from  the  second  step  of  our  systematic  overview  of  usability  evaluator  free- 
responses  identified  two  “system  critical”  issues  relevant  to  the  Consistency  dimension  of 
our  survey.  These  issues  require  specific  fixes  to  ensure  system  optimization.  The  first 
issue  is  in  regards  to  the  “save/next”  buttons  underlying  the  fields  being  filled  out.  These 
buttons  are  not  consistent  from  one  field  to  the  next.  It  was  suggested  that  these  buttons 
be  formatted  the  same  for  every  field.  The  second  issue  is  in  regards  to  the  format  of 
time.  In  the  “Announcements”  field,  military  time  is  requested,  but  civilian  time  shows  up 
after  the  time  is  entered.  The  suggestion  is  to  stick  with  a  single  format  for  time  and,  in 
this  context,  military  time  would  be  most  appropriate.  Again,  these  two  issues  should 
receive  immediate  attention  by  program  developers. 

Finally,  it  is  important  to  note  that  while  our  usability  report  using  the  EQUATE 
uncovered  both  general  and  specific  deficiencies  regarding  PMATT-TA  Increment  1 , 
positive  elements  were  also  identified.  The  value  of  this  information  may  not  be  relevant 
to  the  current  version  of  the  system  but  should  be  incorporated  in  subsequent  versions. 
Specifically,  the  quantitative  analysis  demonstrated  that  only  two  of  our  heuristic 
dimensions  (i.e.,  Learnability  and  Help)  were  below  the  threshold  (i.e.,  less  than  three) 
for  adequacy  in  terms  of  usability.  This  implies  further  attention  from  program  developers 
may  be  necessary  to  achieve  system  optimization.  The  other  six  dimensions  maintained 
averages  that  were  deemed  as  sufficient.  This  simply  implies,  that  for  the  purposes  of 
PMATT-TA,  Error  Handling  &  Feedback,  Graphic  Design  &  Aesthetics,  User  Interaction 
Control,  Memorability  &  Cognitive  Facilitation,  User  Efficiency  and  Consistency,  may  not 
need  the  same  level  of  attention  as  Learnability  and  Help.  In  addition  to  the  qualitative 
analysis,  many  positive  themes  surfaced  in  the  free  responses  from  usability  evaluators. 
We  did  not  systematically  review  positive  comments  as  these  would  not  provide  useful 
information  to  system  developers  for  the  current  version  of  the  system.  However,  a  full 
list  of  the  free  responses  is  provided  in  Appendix  B. 
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8.  Appendices 


Appendix  A. 

PMATT-TA 

Tasks  for  the  Post  Mission  Assessment  for  Tactical  Training  -  Trend  Analysis 

(PMATT-TA) 

Directions:  Below  you  will  find  6  tasks  with  supporting  information.  The  information 
provided  may  not  include  all  required  data ,  so  please  use  random  sample  data  of  your 
choice  in  these  instances.  These  tasks  can  be  accomplished  using  the  web-based 
application  PMATT-TA,  which  can  be  found  at  https://pmatt.innovasi.com.  Please 
complete  each  task  in  full.  When  you  have  completed  all  of  the  tasks,  please  let  the 
experimenter  know. 

1.  The  PMATT-TA  system  will  be  managed  by  the  Command  Patrol  Reconnaissance 
Group,  CPRG.  Part  of  this  responsibility  includes  maintaining  control  over  user 
accounts  and  permissions.  Login  to  the  system  as  CPRGUser  (password: 
Passwordl  I).  Create  a  new  user  account  for  John  Doe,  who  is  a  part  of  the  CPRG 

community  (User  Name _  Password _ ). 

Update  his  profile  to  indicate  that  he  is  assigned  the  role  of  a  CPRW  Staff  Officer. 
After  confirming  the  new  user  account  has  been  created,  log  out  of  PMATT-TA. 

2.  Users  of  PMATT-TA  may  be  required  at  times  to  edit  their  profile.  Login  to  the 
system  as  John  Smith,  the  account  you  just  created.  After  CPRG  created  your 
account,  you  were  assigned  to  the  VP-5  Squadron  and  given  assigned  a  second 
role:  Squadron  Training  Officer.  Update  your  user  profile  with  this  information. 
Additionally,  because  of  these  new  assignments,  you  would  like  to  upload  a  new 
image  to  be  used  for  your  user  profile  on  your  home  page  (use  a  photo  or  icon  of 
your  choice).  After  making  these  updates  to  your  profile  return  to  the  home  screen. 

3.  As  you  are  new  user  of  PMATT-TA,  take  a  few  minutes  to  review  the  tabs  and  menu 
options  available  from  the  home  page.  Use  this  time  to  get  a  general  feel  for  the 
system  to  identify  functionality  that  might  be  required  to  complete  future  tasks.  You 
can  navigate  away  from  the  home  page,  but  please  do  not  save  any  changes  you 
make.  When  you  are  finished  investigating  different  menu  options  and  pages,  please 
return  to  the  home  page  before  continuing  to  task  #4. 

4.  You  have  been  asked  to  create  a  new  event  record  to  document  what  occurred 
during  a  training  event  that  was  conducted  today.  The  P-3C  crew  completed  a  5 
hour  Anti-Submarine  Warfare  (ASW)  simulator  training  event.  This  P-3C  crew  is 
from  VP-5,  CAC-1,  which  documents  training  events  with  a  “T”  Exercise  Indicator. 
The  P-3C  crew  began  their  simulated  event  at  NAS  Pensacola  today  at  0900, 
returning  at  1405. 

The  crew  provides  you  with  the  following  information  during  their  post  mission 
review.  Please  remember  that  this  information  may  not  include  all  required  system 
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data.  If  you  are  missing  required  information,  use  random  sample  data  of  your 
choice. 

EXER/OPER:  This  event  was  part  of  the  exercise  TRITON. 

MSGID:  The  event  was  assigned  serial  number  001  as  part  of  TRITON. 

MISSN:  The  crew  performed  an  Anti-Submarine  Warfare  (ASW)  training  event. 

EVENTINFO:  This  crew  logs  all  events  under  Unit  Designator  VP-5  CHARLIE.  The 
crew  logged  their  actual  and  briefed  event  times  as  indicated  in  the  table  below. 


Event 

Actual  Time 

Briefed  Time 

Take-Off 

0900 

0900 

On-Station 

1000 

0945 

Off-Station 

1245 

1300 

Landing 

1405 

1400 

ALT :  Input  the  On-Station  Altitude  for  the  event  the  crew  recorded  within  their  logs. 

ACFTID:  The  aircraft  for  this  event  was  a  P-3C  with  an  ID  number  of  WTT3. 

AREA:  The  crew  reported  to  area  BRAVO  as  outlined  in  their  briefing. 

LINK:  During  the  event,  the  crew  attempted  to  use  LINK  to  help  communicate  with 
other  aircrews.  It  took  approximately  1  hour  to  establish  LINK  using  High 
Frequency/Net  Control.  The  LINK  remained  in  use  for  3  hours. 

CREW:  The  crew  for  this  event  (crew  number  01 )  included  a  Pilot  (PC,  John  Doe  1 ) 
and  2  sensor  operators  (SSI ,  John  Doe  2;  SS2,  John  Doe  3). 

MET:  The  meteorological  (MET)  report  was  taken  at  1030,  while  at  lat/lon  9999S- 
99990.  This  report  indicated  light  turbulence  with  some  fog. 

CONTACTS:  During  the  event,  the  crew  identified  one  contact  to  report,  an  enemy 
submarine  (serial  number  XXX)  that  remained  stationary  (Lat/Lon  1234S-87654) 
from  1 1 :00  to  1 1 :45. 

NARR:  The  crew  has  requested  that  you  check  the  narrative  (NARR)  for  accuracy 
because  their  Commanding  Officer  reviews  this  part  of  the  report  to  maintain 
awareness  of  crew  events. 

After  entering  this  information,  you  can  log  off  of  PMATT-TA. 

5.  John  Smith  has  reported  to  the  CPRG  community,  as  a  CPRW  Staff  Officer,  that 
several  recent  events  have  been  added  to  the  PMATT-TA  system  and  is  ready  for 
review.  Login  to  the  system  as  CPRGUser  (password:  Passwordl !)  and  go  to  the 
Events  page  to  review  all  events.  CPRG  has  learned  that  a  recent  event  was 
incorrectly  input  in  the  system  and  needs  to  be  deleted.  Delete  event 
_ from  PMATT-TA.  You  have  been  asked  to  learn 


17 


NAWCTSD-SR-201 4-0001 


more  about  what  options  PMATT-TA  provides  for  sorting  and  filtering  events.  Spend 
a  few  minutes  investigating  what  options  are  available. 

6.  CPRG  has  completed  a  review  of  the  events  that  were  part  of  the  TRITON  exercise. 
VP-5  demonstrated  proficiency  during  the  exercise.  Create  a  Bravo  Zulu 
announcement  to  recognize  the  event.  Return  to  the  home  page  to  check  the  display 
of  the  announcement. 

Log  off  PMATT-TA. 
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This  page  intentionally  left  blank. 
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Appendix  B. 

Participant  Usability  Free  Responses 

Graphic  Design  &  Aesthetics 

•  I  found  graphics  and  aesthetics  to  be  pleasing,  clean,  and  directive. 

•  Color,  font,  and  graphics  are  clear  and  readable  with  sufficient  contrast. 

•  (+)  Nice  use  of  graphical  rep  of  events  completed/quais  passed  on  homepage. 

•  (+)  Use  of  red/yellow/green  indicators  for  qualifications  pass/fail  (visually 
appealing). 

•  (+)  Use  of  green  checkmarks  in  "mandatory/opt/misc"  sets  checklist 

•  Not  as  appealing  to  the  eye,  dull  colors,  but  not  a  cumbersome  display. 

•  At  times  it  was  difficult  to  locate  items  in  a  long  drop-down  quickly. 

•  Intuitive  display. 

•  Flag  menu  for  NEW  CONTACT'S  contents  were  red  (text)  not  sure  why  this  is 
useful. 

Overall,  I  found  the  system  design  enhanced  the  user  experience. 

•  I  thought  the  layout  was  clear  and  concise. 

•  The  use  of  colors  and  shading  assisted  in  guiding  me  through  the  task. 

•  I  did  NOT  like  the  latency  within  the  system  however  all  the  overall  inability  to 
search  through  events  drove  me  to  find  creative  ways  to  accomplish  tasks  only 
after  becoming  frustrated  while  sifting  through  1000s  of  events. 

•  The  order  with  which  information  was  given  in  instructions  didn't  clearly  follow 
order  of  vertical  tabs. 

•  There  was  too  much  lag  between  action  and  reaction. 

•  Grouping,  chunking  &  alignment  would  help  guide  user  and  organize  related 
input. 

•  There  was  not  much  clutter,  only  found  difficulty  when  in  a  long  drop-down  field. 

•  Had  some  admin  issues  in  the  beginning,  but  had  no  issues  afterwards. 

•  Overall  the  system  was  clean/not  cluttered. 

•  Design/aesthetics  were  minimalist  and  functional.  Given  the  intended  audience,  it 
seemed  appropriate. 

•  Updating  profile  -  location  more  visible. 

•  Roles  of  user  were  not  clear  to  what  actions  they  could  person  on  the  system. 

Overall,  I  found  the  layout  to  be  visually  appealing. 

•  The  tabs  (data  entry  boxes)  were  not  grouped  in  a  way  that  told  me  that 
information  within  them  was  related. 

•  Instructions  were  confusing  and  seemingly  out  of  order  relative  to  interface. 

•  Alignment  of  grouped  input  would  enhance  understanding  of  functional 
relationships. 

•  Although  the  "New  Event"  button  is  on  the  home  page,  it  blends  in  somewhat 
with  everything  else.  Consider  increasing  the  buttons  size/color/highlight  etc. 

•  Under  "Administration  >  Rules  Engine"  images  aren't  all  loading. 
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•  Was  not  able  to  "reports"  tab;  needed  additional  access. 

•  Consistency  issue  with  entering  time.  Stick  to  one  method.  Prefer  just  typing  it 
in  and  have  a  guide  for  format 

•  I  think  menu  contents/text  should  be  same  color  across  all  menus  (preferably 
black)  should  not  change  based  on  conditions  (e.g.,  error  state). 

•  Most  error  messages  appeared  in  lower  portion  of  window,  but  one  error 
message  appeared  in  upper  portion. 

Overall,  I  found  characteristics  of  the  graphic  interface  (e.g.,  graphics,  text,  colors) 
supported  a  positive  user  experience  by  increasing  usability. 

•  I  thought  the  layout  was  clear  and  concise. 

•  The  use  of  colors  and  shading  assisted  in  guiding  me  through  the  task. 

•  I  did  NOT  like  the  latency  within  the  system  however  all  the  overall  inability  to 
search  through  events  drove  me  to  find  creative  ways  to  accomplish  tasks  only 
after  becoming  frustrated  while  sifting  through  1000s  of  events. 

•  The  only  graphic  I  recall  was  the  map  of  Earth,  but  no  reason  for  this  picture  was 
given. 

•  The  system  was  unpleasant  to  use  due  to  the  long  wait  times  and  processing 
times. 

•  Too  many  components  within  certain  tabs. 

•  Need  more  navigation  support.  The  order  wasn't  implicit  enough  by  tabs.  Need  to 
be  explicit  to  see  what  has  to  be  done  first  and  when  complete. 

•  The  lag  in  the  system  made  me  think  a  button  wasn't  active  and  I  would  either 
click  again  or  click  somewhere  else.  The  system  would  go  from  no  action  to  a 
burst  of  several  actions  and  then  I  would  have  to  figure  out  what  did/did  not  just 
occur. 

•  I  found  the  interaction  confusing,  hard  to  follow,  and  unstructured.  May  have 
been  a  function  of  unfamiliarity  with  the  context  or  a  shortfall  within  the 
instructions. 

•  Under  the  "Link"  set,  a  "[CHECK]"  and  "X"  response  box  set  is  used  but  isn't 
used  anywhere  else  in  the  system  up  to  that  point. 

o  Under  the  "Link"  summary,  both  the  column  for  "Hours  Attempted"  and 
"Hours  Successful"  are  labeled  "HOURS"  only  so  you  can  not  differentiate 
the  two. 

o  Button  to  move  to  next  set  labeled  "NEXT"  instead  of  "Save  Changes  and 
Next"  like  every  other  set. 

•  Graphical  picture  buttons  ([CHECK],  X,  Pencil)  functions  would  be  clearer  if  a 
small  popup  appears  once  you  hover  your  cursor  above  each  graphic  in  which  its 
function  is  described. 

•  Sorting  isn't  intuitive  in  drop-downs:  1,10,11,12,2,3,4  instead  of  1 , 2,  3,  4,  1 0, 

11,  12 

•  Graphics  were  not  used  often,  which  could  have  added  to  the  visual  aspects  of 
the  system. 

•  The  addition  of  more  colors  can  help  discriminate  between  tabs. 
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•  Would  be  nice  to  clearly  label  required  fields  (not  just  red  error  messages  when 
you  try  to  save). 

•  Not  many  graphics  to  judge,  which  isn't  a  bad  thing. 

•  "[CHECK]"  and  "X"  sometimes  used,  but  not  throughout 

•  One  time  the  high  level  menu  disappeared,  but  was  able  to  find  home  again 
quickly 

•  Entering  time  was  inconsistent.  Too  many  different  ways  (scroll,  input  manually, 
use  two  side  scrolls).  Just  make  manual  with  consistent  format. 

•  Not  a  clear  path  for  where  you  are  within  the  options/menus.  Maybe  add  path  at 
top  as  is  common  in  Windows.  "Home/events"  also  makes  it  easy  to  go  back. 

•  Liked  the  check  names  in  the  side  bar. 

•  The  additional  info  bubbles  could  be  more  clear  -  maybe  a  link  titled  "more  info" 
to  make  it  clear  that  there  is  more  info.  Many  times  could  not  get  the  pop  ups  to 
show  just  by  moving  the  mouse  over  it. 

Error  Handling  &  Feedback 

•  The  error  handling  and  feedback  was  executed  well  in  my  opinion.  I  was  able  to 
very  quickly  once  presented,  understand  and  correct  any  said  error. 

•  Red  boxes  and  alerts  make  it  easy  to  identify  missed  steps. 

•  Feedback  only  provided  after  save. 

•  Required  fields  not  marked. 

•  (-)  Use  of  processing  message  to  indicate  system  progress. 

•  (+)  incorrect  formatting  of  response  in  "New  Event"  indicated  by  red  highlighting. 

•  (+)  Acknowledgement  of  successful  processing. 

•  (+)  Times  turned  red  for  future  times  when  entering  supposedly  past  date. 

•  Error  messages  were  descriptive  enough  to  help  me  resolve  the  problem. 

•  Gave  me  error  and  told  me  to  enter  email  address  (clear  message). 

•  EVENT  INFO,  UnitDesignator  field,  I  typed  in  the  example  provided.  An  error 
msg  appeared  at  top  of  screen  and  disappeared  before  I  read  it.  Then  all  data  I 
had  entered  reverted  to  default  values. 
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Overall,  I  found  the  system  design  prevented  errors  and  unintended  actions. 

•  The  system  provided  error  correction  feedback  through  red  text  and  impeded  my 
advancement  and  cascading  of  errors.  I  thought  this  feature  was  executed  well 
sans  the  delay. 

•  System  does  not  indicate  (under  "New  Event")  which  responses  are  mandatory 
until  AFTER  you  attempt  to  save  your  responses.  It  would  be  better  to  include  a 
visual  indicator  (i.e.,  asterisk)  next  to  questions  requiring  a  forced  response. 

•  Long  lag  time  sometimes  created  the  impression  that  the  system  was  "frozen" 
especially  on  homepage. 

•  Some  fields  involved  in  creating  a  new  event  did  not  always  give  you  examples 
of  what  should  go  in  the  field,  but  instead  would  just  remain  red  until  you 
changed  the  input. 

•  Within  that  pop  up  screen,  did  not  allow  you  to  move  beyond  pop  up  screen  until 
error  was  corrected. 

•  When  deleting,  it  asked  if  I  was  sure,  but  didn't  ask  if  I  wanted  to  save  an  entry 
before  moving  on  and  it  did  not  auto  save. 

•  When  error  messages  would  pop  up  they  were  displayed  too  quick;  needed  more 
explanation. 

•  Didn't  know  the  autosave  was  an  option. 

•  Once  I  pressed  the  backspace  bar  on  the  keyboard  and  it  closed  out  the  "create 
event"  window.  I  had  to  type  in  the  info  again. 

Overall,  I  found  the  system  provided  informative  warnings/alerts  when  errors  were 
identified. 

•  The  system  provided  error  correction  feedback  through  red  text  and  impeded  my 
advancement  and  cascading  of  errors.  I  thought  this  feature  was  executed  well 
sans  the  delay. 

•  Required  data  that  wasn't  provided  or  improperly  formatted  input  data  was  not 
presented  through  an  error  message  until  the  save  and  continue  action  was 
selected.  Knowing  inaccuracies  along  the  way  would  be  helpful. 

•  Messages  indicated  a  field  needed  to  be  filled,  but  not  the  type  of  thing  to  fill  it 
with.  Did  give  specific  error  when  I  entered  time  wrong  though. 

•  Error  messages  just  said  that  the  info  for  that  box  was  required  but  I  didn't  know 
what  info  to  put. 

•  It  wasn't  clear  what  info  was  required  in  the  box.  It  seemed  to  accept  anything. 

Overall,  I  found  the  system  provided  appropriate  feedback  when  interacting  with  the 
system  (e.g.,  error  message,  type  of  input). 

•  I  did  however  the  latency  was  unsat. 

•  Button  clicks  were  acknowledged,  but  then  nothing  until  the  screen  refreshed. 

•  I  did  not  have  appropriate  permission  to  view  records  but  was  unaware  of  this 
restricted  permission  and  thought  it  was  a  glitch. 
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•  Feedback  was  provided  when  a  tab  was  complete  enough  to  continue,  but  I 
didn't  know  which  components  were  required  fields  until  I  selected  Save  and 
Continue. 

•  Didn't  see  status  indicators. 

•  There  were  times  when  I  would  select  something  and  see  no  indication  of 
application  or  progress.  I  would  select  again  or  try  another  function.  I  couldn't  tell 
if  it  didn't  work  or  was  locked  up. 

•  On  home  page,  long  lag  between  "no  records  to  display"  notification  and  actual 
records  appearing.  It  would  be  better  for  results  box  to  just  stay  blank  until  results 
appear,  otherwise  it  can  be  misleading  at  first  glance. 

•  Completed  tasks  were  usually  marked  by  a  check  and  next  field  would  pop  up 
after  I  completed  the  sections  by  pressing  Save  &  Next. 

•  When  going  to  a  new  section,  the  processing  icon  is  displayed  to  let  me  know  the 
status  of  the  system. 

•  Spinning  timer  used,  not  really  good  indicator.  Progress  bar  preferred  so  you 
can  tell  if  it  gets  stuck/freezes. 

User  Interaction  Control 

•  The  controls  and  interface  were  very  straightforward  from  a  data  entry  standpoint 
however,  I  was  disappointed  from  the  account  management  standpoint  as  a  non¬ 
administrator  (we  were  able  to  log  in  in  two  different  methods  initially). 

•  Lag  in  input  &  reaction. 

•  (-)  No  ability  to  sort  each  column  on  home  page. 

•  (+)  Sorting  &  Filtering  IS  available  in  events  tab  (see  consistency). 

•  (-)  Only  able  to  select  time  in  "New  Event"  with  Sliders. 

•  I  had  complete  control  of  the  info  that  I  placed  within  the  field  and  the  system 
allows  me  to  make  edits  to  events  needed  in  a  quick  manner. 

•  Calendar  slide  control  does  not  fully  support  clicking  on  desired  number. 

•  After  correcting  landing  briefed  time,  box  remained  red. 

Overall,  I  found  the  system  made  the  user  feel  in  charge  by  responding  to  user  actions 

(e.g.,  acknowledge  button  clicks,  indicate  wait  times). 

•  Button  activation  not  apparent  and  wait  times  not  obvious. 

•  There  were  items  that  caused  an  activation  instead  of  simple  selection.  Undo 
would  be  helpful  (selecting  an  event). 

•  No  ability  to  sort  each  column  on  home  page,  but  possible  in  other  windows  (like 
events  tab).  Make  all  columns  in  interface  sortable. 

•  I  felt  like  I  had  complete  control  of  the  system. 

Overall,  I  found  the  system  responded  to  user  actions. 

•  I  did  however  the  latency  was  unsat. 

•  I  tried  to  use  the  ?  Icons  several  times  to  determine  required  format  and/or 
content.  Sometimes  it  was  helpful  and  sometimes  not  at  all.  Additional  info  would 
have  been  helpful. 
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•  No  "help"  that  was  accessible  at  all  times.  Some  boxes  had  but  not  all. 
Should  be  consistent  and  some  helps  should  always  be  available  with  the  ability 
to  search  topics. 

•  Did  not  know  had  to  change  the  role  to  change  a  profile. 

Overall,  I  found  the  system  provided  flexibility  to  change  features  of  the  interface  (e.g., 
reconfigure  windows,  modify  toolbars). 

•  It  did  not.  I  did  not  see  any  flexibility  within  the  system  nor  do  I  think  there  should 
be.  It  serves  a  very  specified  purpose  and  besides  the  latency  it  appears  to  suit 
that  purpose  well. 

•  Inconsistent  date  formatting  across  interface  (mm/dd/yyyy  vs  ddmmmyyyy). 

•  Unable  to  sort  all  columns. 

•  It  was  not  flexible  from  my  perspective.  I  felt  I  had  to  leave  things  how  they  were. 
This  is  not  a  big  deal  to  me. 

Overall,  I  found  the  system  easy  to  navigate. 

•  Sometimes  I  didn't  realize  they  were  there  and  if  I  don't  see  them,  they  don't  exist 
(e.g.,  tabs,  options). 

•  "Reports"  section  was  not  accessible. 

Memorability  &  Cognitive  Facilitation 

•  I  did  not  see  any  need  to  offload  as  all  the  data  was  hardcopy  presented. 

•  Check  marks  show  which  fields  have  already  been  entered,  if  not  necessarily 
completed. 

•  (-)  Under  "Link"  set,  a  "check"  and  "X"  dialog  box  set  is  used  but  isn't  used 
anywhere  else  in  the  system  up  to  this  point. 

•  (-)  Under  the  link  summary,  both  the  column  for  "hours  attempted"  and  "hours 
successful"  are  labeled  "HOURS"  only. 

•  (-)  Button  to  move  to  next  set  labeled  "NEXT"  instead  of  "Save  changes  &  Next" 
like  every  other  set. 

•  The  design  of  the  system  makes  it  easy  to  navigate  back  and  forth  except  for  the 
reports  tab.  Could  not  access  without  additional  approval.  I  feel  familiar  enough 
with  the  system  to  complete  the  additional  tasking. 

•  Lots  of  acronyms. 

Overall,  I  found  the  system  provided  easily  understandable  information  in  a  format  that 
minimized  cognitive  workload. 

•  The  error  checking  was  incredibly  beneficial  and  allowed  me  to  immediately 
address  the  problem. 

•  Did  not  find  option  for  New  User  until  searching  several  screens. 

•  Unable  to  know  which  information  is  located  in  a  specific  event  sub-tab. 

•  Difficult  to  know  what  event  info  goes  where,  so  I  just  clicked  every  sub-menu 
and  re-read  the  script  to  see  if  any  info  fit.  Rather  tedious  but  I  expect  actual 
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users  will  know  where  the  info  belongs?  If  not,  perhaps  a  mouse-over  or  preview 
of  the  info  asked  for  would  be  helpful. 

•  Units  of  measurement  for  input  boxes  would  have  been  helpful  as  a  prompt  for 
identifying  correct  input  data. 

•  Grouping  of  information  in  a  manner  that  implied  sequence  and  relation  would 
help. 

•  Altitude  was  addressed  in  more  than  one  tab. 

•  Instructions  were  unclear  and  hard  to  follow  certain  parts  would  have  been  much 
easier  with  substeps  listed  out. 

•  I  was  confused  whether  ID  number  was  same  as  mission  name. 

•  Under  “MISSN,”  all  acronyms  are  explained  except  for  those  under  "Purpose" 
which  is  actually  one  of  only  2  required  fields. 

•  Under  "New  Link"  >  "Transmission  Mode"  full  acronym  descriptors  are  cutoff, 
making  selection  difficult  unless  you  have  an  acronym  guide  available  externally. 
Consider  increasing  dropdown  width  or  decreasing  font  size. 

•  Sometimes  information  was  showed  in  "?"  to  know  what  inputs  were  needed,  but 
not  always.  Make  help  consistent  and  available. 

•  Steps  were  not  exactly  given.  Input  could  be  anywhere  at  anytime.  This  is  good 
for  flexibility,  but  makes  it  a  little  confusing  for  navigating.  I  wouldn't  change  it 
though.  I  think  it  would  be  a  user  preference  thing. 

•  I  was  unfamiliar  with  the  info  being  entered  so  I  didn't  know  what  info  went  into 
which  box. 

Overall,  I  found  the  system  provided  aids  that  helped  with  memory  limitations. 

•  Check  marks  show  which  fields  have  already  been  entered,  if  not  necessarily 
completed. 

•  Unable  to  know  which  information  is  located  in  a  specific  event  sub-tab. 

•  Mouse-over  question  marks  were  helpful. 

•  Graphics  in  instructions  would  be  helpful. 

•  Instructions  were  confusing  and  hard  to  understand. 

•  Directions  were  not  written  in  steps  at  all  and  were  confusing. 

•  The  use  of  learning  aids  at  all  was  not  apparent. 

User  Efficiency 

•  I  thought  the  design  and  system  was  very  clean  (sans  latency)  and  it  supported 
the  event  logging  task  well. 

•  Did  not  find  option  for  New  User  until  searching  several  screens. 

•  Unable  to  read  announcement  (after  clicking  Admin>Announcements)  without 
opening  Edit  dialogue  box. 

•  Unable  to  know  which  information  is  located  in  a  specific  event  sub-tab. 

•  (-)  Long  lat  time  on  home  page  for  events/quals  to  populate. 

•  (-)  Long  lag  between  "no  records  to  display"  and  actual  records  appearing. 

•  (-)  "New  Event"  button  available  from  home  page  but  could  stand  out  better. 
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•  The  design  was  easy  to  use  and  with  proper  instructions,  locating  fields  to  be 
populated  was  not  difficult. 

•  When  switching  between  tabs  (user  details  and  roles)  should  ask  if  you  want  to 
save  so  you  don't  waste  time  re-entering  data  if  you  forgot  to  save. 

•  MSID  Originator  ?  Mouse-over  info  disappears  too  quickly. 

•  Slider  bar  for  time  entry  for  NEW  CONTACT  is  not  easy  to  use  (moves  thru  times 
too  quickly) 

Overall,  I  found  the  design  supported  completion  of  tasks  with  minimal  time  and  effort. 

•  Latency. .  ,(x3)  this  was  by  far  the  most  significant  and  annoying  downfall  of  the 
design  of  the  system. 

•  Difficult  to  know  what  event  info  goes  where,  so  I  just  clicked  every  sub-menu 
and  re-read  the  script  to  see  if  any  info  fit.  Rather  tedious  but  I  expect  actual 
users  will  know  where  the  info  belongs?  If  not,  perhaps  a  mouse-over  or  preview 
of  the  info  asked  for  would  be  helpful. 

•  I  kept  getting  stuck  on  steps  that  seemed  like  they  should  be  easy. 

•  Instructions  hard  to  follow  and  extract  data. 

•  Hard  to  know  what  information  was  required. 

•  The  ?  Button  wasn't  able  to  reselect;  dialog  box  closed  too  quickly. 

•  Seemed  like  input  could  have  been  chunked. 

•  No  search  bar  within  system,  but  after  a  few  trial  runs  the  location  of  all  features 
should  become  more  apparent  anyway. 

•  No  search  options.  Should  add  to  main  screen  with  help. 

•  Complex  task  required  filling  in  multiple  tabs,  but  it  was  easy  to  go  back  to  old, 
just  more  time  consuming. 

Learnability 

•  I  found  the  system  intuitive  from  a  user  standpoint,  however  from  an  admin 
standpoint  I  struggled  as  permissions  and  menu  structure  for  those  management 
functions  were  not  implemented  smoothly. 

•  (-)  Under  EVENTS  tab,  commands  column  >  pencil  box  &  "X"  box  (I  don’t  know 
what  they  represent,  perhaps  have  a  descriptor  appear  when  you  hover  over 
each  item). 

•  The  functions  within  the  system  allows  for  learning  to  occur. 

•  Buttons  mostly  self  explanatory. 

Overall,  I  found  the  system  easy  to  learn  how  to  use. 

•  I  found  the  system  very  intuitive  once  I  was  able  to  maneuver  through  the  latency 
issues. 

•  Difficult  to  know  what  event  info  goes  where,  so  I  just  clicked  every  sub-menu 
and  re-read  the  script  to  see  if  any  info  fit.  Rather  tedious  but  I  expect  actual 
users  will  know  where  the  info  belongs?  If  not,  perhaps  a  mouse-over  or  preview 
of  the  info  asked  for  would  be  helpful. 

•  Vertical  tabs  were  unfamiliar. 
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•  Could  be  emphasized  by  my  lack  of  familiarity  with  the  context  and  platform 
[system  would  be  difficult  for  a  new  user  to  use]. 

•  Some  operational  background  would  have  increased  understanding  of  data  entry 
and  data  relationships. 

•  Step-by-step  instructions  like  those  given  were  very  helpful.  Without  them,  simply 
figuring  out  how  to  navigate  and  properly  complete  forms  could  be  challenging. 
As  mentioned  elsewhere,  consider  some  form  of  guided  tutorial. 

•  Not  much  training  is  needed  to  understand  and  use  the  system.  It  is  only 
important  to  know  the  terminology. 

•  Terms  and  words  used  were  not  commonly  known. 

Overall,  I  found  the  embedded  training  (e.g.,  tutorials,  help)  supported  learning  how  to 
use  the  system. 

•  The  mouse-over  question  marks  were  helpful  but  there  were  no  images 
otherwise  in  them. 

•  The  ?  Buttons  closed  too  quickly  and  didn't  always  provide  clarifying  information. 

•  Learning  aids,  if  available,  weren't  present.  However,  having  guilt  in  tutorials  for 
each  activity  would  be  helpful,  especially  for  those  with  little  previous  exposure  to 
these  types  of  systems. 

•  Did  not  notice  a  tutorial  but  the  help  features  did  assist  in  use  of  the  system. 

•  No  "help"  or  "how  to  use  system." 

•  The  "?"  entries  were  long  and  needed  to  be  assessed  several  times.  They 
weren't  always  specific  enough  to  tell  you  how  to  enter  data  either.  For  example, 
didn't  always  give  measurement  unit  to  use  (ft,  etc.) 

•  Learning  aids  were  not  clear  on  how  to  obtain  them. 

Consistency 

•  The  system  appeared  to  maintain  consistency,  again  in  the  user  functions  but 
NOT  in  the  admin  functions. 

•  User  term  'sort'  not  'group'. 

•  (-)  Inconsistent  date  formats  (mm/dd/yyyy  vs  ddmmmyyyy) 

•  Yes  system  formatting  is  consistent. 

•  Administration  tab  seemed  a  good  place  for  user  creation. 

•  Why  is  the  time  entry  a  sliding  bar?  Weird  and  unusual!  A  simple  typed  interface 
is  easiest  and  most  consistent  with  way  things  are  normally  done. 

•  Inconsistent  save/next  buttons/GUI  for  LINK  entry. 

•  For  MET  date/time  I  had  to  manually  enter  a "/"  between  the  date  and  time. 

•  For  CONTACTS,  methods  to  input  data  and  time  inconsistent  with  other  sets  - 
1  st  time  I  click  on  start  time  entry  box,  the  drop-down  list  appeared.  2nd  time  I 
clicked  on  box  to  fix  an  error  only  the  cursor  appeared  (no  drop-down). 

•  To  filter  events  by  date  when  I  click  in  the  date  field  cursor  appears,  but  calendar 
does  not  pop-up  (inconsistent  with  other  date  entry  fields). 

•  When  entering  new  announcement,  drop-down  list  for  time  entry  is  in  civilian 
format  (5:00PM),  but  after  selection  is  displayed  in  military  format  (1700). 
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Overall,  I  found  the  system  maintained  consistency  (e.g.,  layout,  actions,  formatting). 

•  Military  time  not  used  in  narrative. 

•  Date/time  &  button  titles/images  vary  throughout  interface.  Consider  further 
standardization. 

•  Abbreviations  not  always  explained  due  to  space  limitations. 

•  Some  columns  were  sortable/searchable  while  others  equally  important  weren't. 
Consider  changing  all  columns  to  sorting  if  applicable  and  appropriate. 

•  No  consistent  use  of  time  entry,  saving  buttons,  where  to  find  buttons,  etc. 

•  "[CHECK]"  and  "X"  are  not  well  known  ways  to  save  and  cancel  (especially 
"[CHECK]").  Stick  to  words  or  use  floppy  save  icon. 

•  Tasks  required  me  to  entered  the  time  as  1 405,  and  I  was  able  to  do  so,  but  the 
system  later  showed  the  time  entered  was  205  (task  #4  -  Event  Info). 

Help 

•  I  found  the  help  dialogue  bubbles  where  very  easy  to  use  and  I  did  rely  on  them 
a  few  times. 

•  Did  not  see  a  help  menu. 

•  Mouse-over  question  marks  were  helpful. 

•  Open  too  short  and  couldn't  access  again. 

•  (+)  Question  mark  popups  with  instructions  -  good  use  of  example  responses. 

•  ?  Button  clears  out  too  quickly  after  placing  the  mouse  over  it  (makes  it  difficult  to 
read  entire  advice. 

•  No  help  option  -  need  one  to  explain  options,  etc.  Has  some  "?"  next  to  items,  but 
no  general  "help"  button. 

•  General  -  disappearing  tool  tips. 

Overall,  I  found  the  help  references  easy  to  use  to  support  task  completion. 

•  The  help  pop  ups  provided  a  lot  of  utility  and  help. 

•  There  was  no  help  file  or  document  to  search  that  I  saw. 

•  Closed  too  quickly;  not  always  clarifying.  When  they  told  me  required 
characteristics  of  data,  it  was  very  helpful  (e.g.,  number  of  letters;  units  of 
measurement;  definition). 

•  It  was  hard  to  figure  out  what  information  went  where  from  instruction  sheet. 

•  I  had  to  re-read  and  read  ahead  to  identify  appropriate  information  to  extract  and 
use. 

•  Boxes  closed  too  quickly.  See  above. 

•  Instructions  were  difficult  to  understand. 

•  No  general  help/FAQ  section  was  observed,  although  this  may  have  been  an 
oversight. 

•  Help  function  were  located  next  to  most  fields.  Helped  to  understand  what  kind  of 
info  was  needed  for  the  field. 

•  No  general  help  given.  No  search  capabilities  either.  Highly  suggest  adding 
both. 

•  Did  not  see  any  instructions  provided  by  the  system  besides  error  messages. 
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Miscellaneous 

•  Difficult  to  know  what  event  info  goes  where,  so  I  just  clicked  every  sub-menu 
and  re-read  the  script  to  see  if  any  info  fit.  Rather  tedious  but  I  expect  actual 
users  will  know  where  the  info  belongs?  If  not,  perhaps  a  mouse-over  or  preview 
of  the  info  asked  for  would  be  helpful. 

•  Needed  units  of  measurement  prompt  input. 

•  Didn't  know  what  path  or  what  to  do  next. 

•  Why  is  narrative  not  in  military  time? 

•  Could  not  select  event  without  opening  it  and  no  progress  indicator. 

•  Have  drop  down  options  in  filter  boxes. 

•  Allow  drag  and  drop  columns. 

•  Allow  hide/unhide. 

•  Under  "Administration  >  Rules  Engine"  images  aren't  all  loading. 

•  Very  slow  and  lagging  in  "New  Event"  dialog/popup. 

•  Sorting  isn't  intuitive  in  drop-downs  (1,10,11,12,2,3,4...  instead  of  1 ,  2,  3,  4, 

10,11,12). 

•  Nice  use  of  map  in  "Contacts"  set. 

•  Under  "MISSN"  all  acronyms  are  explained  except  for  those  under  "Purpose" 
which  happens  to  be  a  required  field. 

•  Under  "New  Link"  >  "Transmission  Mode"  full  acronym  descriptors  are  cut  off. 

•  Under  "NARR"  (auto-generated  narrative)  CAC  is  placed  twice  in  the  script.  Also, 
includes  "Wing  Commander  Patrol"  &  "Reconnaissance  Wing  Eleven"  who 
weren't  included. 

•  Great  to  have  the  filter/sort  feature  to  search  by  a  specific  event  #. 

•  Announcement  date/time  columns  both  labeled  as  "Announcement". 

•  "[CHECK]"  and  "X"  -  no  good  -  change  to  words...  Is  it  "submit"  and  "cancel".  Not 
completely  intuitive,  also  inconsistent  with  "save  changes,"  "delete"  used  else 
where. 

•  Scrolling  option  for  time  -  VERY  inefficient  and  inconsistent  with  other  save  entry. 

•  Also  used  "insert"  instead  of  "save"  like  other  places 

•  I  added  squadron  on  "user  details"  then  changed  to  "roles"  without  saving.  It 
didn't  change  or  ask  if  I  wanted  to  save. 

•  What  is  the  difference  between  "save  changes"  and  "save  changes  +  next?" 

•  What  is  "export  purple?" 

•  System  uses  way  too  many  short  hands  (abbreviations  and  acronyms).  There  is 
space,  spell  out. 

•  What  is  different  between  "serial  #"  and  "serial  #  of  qualifier?" 

•  Turned  my  future  time  red,  but  didn't  say  why  (did  after  I  submitted/saved). 

•  It's  not  clear  what  is  required  and  optional. 

•  Sometimes  "save"  or  "[CHECK]"  was  on  left,  sometimes  on  right. 

•  I  found  no  assistance  from  the  system  outside  of  the  error  monitoring. 

•  The  system  got  hung  up  a  lot  when  input  on  next  step  was  chosen. 
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•  I  did  not  observe  an  auto-save  feature,  but  I  may  have  missed  it.  If  it  isn't  present, 
it  would  be  useful  to  implement. 

•  "Done  quickly?" 

•  No  auto-save  or  even  a  "are  you  sure  you  want  to  go  on  before  saving" 
message.  Should  add  this  or  "would  you  like  to  save  before  continuing?" 

•  Saving  was  not  automatic. 

•  The  latency  was  unsat. 

•  Needed  navigation  support. 

•  Needed  status  indication. 

•  Improper  inputs  were  identified  after  save  was  executed,  but  would  be  helpful  at 
time  of  input. 

•  Major  lags  and  occurrences  of  hang  ups. 

•  Only  able  to  select  times  in  "New  Event"  with  sliders,  but  in  another  section  times 
are  typable.  Time  inputs  should  be  standardized  one  way  or  another  across  the 
interface. 

•  No  ability  to  sort  each  column  on  home  page,  but  possible  in  other  windows  (like 
events  tab).  Make  all  columns  in  interface  sortable. 

•  Same  as  status  (running  timer)  as  before.  Make  progress  bar  instead. 

•  The  latency  was  the  killer  in  this  application.  It  exceeds  any  best  HF  practices 
and  even  with  fast  connection  speeds  is  painfully  slow. 

•  (-)  Does  not  indicate  under  "New  Event"  details  which  responses  are  mandatory 
until  AFTER  you  attempt  to  save  your  responses. 

•  I  did  experience  an  error  earlier  on  during  task  1  because  the  user  name  I  was 
instructed  to  use  was  already  taken.  Also  my  Bravo  Zulu  was  not  updated  to  the 
home  page  announcement  section. 

•  For  MET,  I  entered  0  cloud  cover,  but  it  required  me  to  enter  a  cloud  type.  I 
entered  cloud  type  and  pressed  [CFIECK]  button  and  error  message  remained. 
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Appendix  C. 

EQUATE  Dimensions  &  Sub-dimensions 


1.  Graphic  Design  &  Aesthetics 


i.  The  system  design  enhanced  the  user  experience. 

ii.  The  layout  [was]  visually  appealing. 

...  Characteristics  of  the  graphic  interface  (e.g.,  graphics,  text,  colors) 
supported  a  positive  user  experience  by  increasing  usability. 


2.  Error  Handling  &  Feedback 


i.  The  system  design  prevented  errors  and  unintended  actions. 

..  The  system  provided  informative  warnings/alerts  when  errors  were 
IL  identified. 

...  The  system  provided  appropriate  feedback  when  interacting  with  the 
system  (e.g.,  error  message,  type  of  input). 


3.  User  Interaction  Control 


.  The  system  made  the  user  feel  in  charge  by  responding  to  user 
actions  (e.g.,  acknowledge  button  clicks,  indicate  wait  times), 
ii.  The  system  responded  to  user  actions. 

...  The  system  provided  flexibility  to  change  features  of  the  interface 
(e.g.,  reconfigure  windows,  modify  toolbars), 
iv.  The  system  [was]  easy  to  navigate. 


Memorability  &  Cognitive  Facilitation 


.  The  system  provided  easily  understandable  information  in  a  format 
that  minimized  cognitive  workload. 

ii.  The  system  provided  aids  that  helped  with  memory  limitations. 


5.  User  Efficiency 


.  The  design  supported  completion  of  tasks  with  minimal  time  and 
L  effort. 


6.  Learnability 


i.  The  system  [was]  easy  to  learn  how  to  use. 

..  The  embedded  training  (e.g.,  tutorials,  help)  supported  learning  how 
to  use  the  system. 


Consistency 


i.  The  system  maintained  consistency  (e.g.,  layout,  actions,  formatting). 


8.  Help 


i.  The  help  references  [were]  easy  to  use  to  support  task  completion. 


Note:  Each  sub-dimension  started  with  "Overall,  I  found...” 
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Appendix  D. 

Formatting  of  EQUATE  Sub-Dimension  with  Sub  items 


Strongly  Strongly  N/A 

Disagree  Agree 

1.)  Overall,  I  found  the  system  design  1  2  3  4  5  0 

enhanced  the  user  experience. 


Never 

Always 

Examples  of  Strenqths  Observed 

The  design  provided  a  pleasant 
experience. 

1 

2 

3 

4 

5 

The  system  design  had  a  professional 
appearance. 

1 

2 

3 

4 

5 

Examples  of  Weaknesses  Observed 

There  was  too  much  clutter  on  the  display. 

1 

2 

3 

4 

5 

The  design  negatively  impacted  the 
overall  experience. 

1 

2 

3 

4 

5 
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9.  List  of  Acronyms 

1 .  EQUATE  -  Experience-based  Questionnaire  for  Usability  Assessment 
Targeting  Elaborations 

2.  HCI-GUI  -  Human-Computer  Interaction  -  Graphic  User-Interface 

3.  PMATT-TA  -  Post  Mission  Assessment  for  Tactical  Training  -  Trends 
Analysis 

4.  SME  -  Subject  Matter  Expert 


3 


